Add the DC-EGM solver (discrete-continuous endogenous grid method)#390
Draft
hmgaudecker wants to merge 203 commits into
Draft
Add the DC-EGM solver (discrete-continuous endogenous grid method)#390hmgaudecker wants to merge 203 commits into
hmgaudecker wants to merge 203 commits into
Conversation
Route the existing brute-force grid search through a per-regime solver configuration and a builder registry, with no change to the numerics. - `Regime.solver: BruteForce | DCEGM` (default `BruteForce()`), exported from `lcm` alongside the `DCEGM` configuration class. - `_lcm.solution.registry`: `SolverBuildContext`, `SolverKernels`, the `SolverKernelBuilder` protocol, and `_build_brute_force_kernels` (the former `_build_max_Q_over_a_per_period`), dispatched on `type(regime.solver)` via `SOLVER_KERNEL_BUILDERS`. - `DCEGM` is published as the final configuration surface (fields + field validation), but its engine is not yet wired in; a regime requesting it is rejected at model build with `NotImplementedError`. Behavior-preserving: the full test suite passes unchanged and an explicit `BruteForce()` yields the same value function as the default. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Points the benchmark feature's aca-model at feat/dcegm-solver's tip — main's audit fixes + the DC-EGM solver + smooth-share eligibility — the version the solver-seam work is developed against. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the `type(solver)`-keyed builder registry with a polymorphic `Solver` ABC. The engine now calls `solver.validate(context)` then `solver.build_period_kernels(context)` — no `SOLVER_KERNEL_BUILDERS` dict, no `BruteForce | DCEGM` union, no standalone DC-EGM guard. - `_lcm/solution/contract.py` (new): the `Solver` ABC (abstract `build_period_kernels`, default no-op `validate`), `SolverBuildContext`, and `SolverKernels`. An engine leaf — imports nothing that reaches `lcm.solvers`, so the façade can re-export it without an import cycle. - `_lcm/solution/solvers.py` (new): `GridSearch(Solver)` (the relocated grid-search builder, with function-local `jax`/`get_max_Q_over_a` imports) and `DCEGM(Solver)` (the published config; `validate` raises the not-yet-available guard, so a regime requesting it is rejected at model build). - `lcm/solvers.py` → thin re-export façade; `registry.py` deleted; the `processing` dispatch and `Regime.solver` field updated. - Rename the default solver `BruteForce` → `GridSearch` (more descriptive; alpha permits the break). Faithful to dcegm-solver-seam-abc-design.md (ABC, not Protocol). Layer 2 (generic KernelResult) does not apply here — the stub seam has no EGM fork. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
Benchmark comparison (main → HEAD)Comparing
|
e0fecdd to
44e8522
Compare
The full discrete-continuous endogenous grid method, re-rooted as a single commit on top of the solver-selection seam (PR #388). The tree reproduces the reviewed feat/dcegm tip exactly, minus two root-level audit scratch files. On top of the seam this adds the `_lcm.egm` engine, DC-EGM build-time validation, the `_build_dcegm_kernels` builder and the EGM-specific SolverBuildContext / SolverKernels / SolutionPhase fields, and replaces the seam's build-time `NotImplementedError` guard with the real solver. To be split into reviewable sub-PRs at review time. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The child stochastic-node expectation built the full node-stacked `node_values`/`node_marginals` arrays and reduced them with a weighted sum downstream of the map. So `stochastic_node_batch_size` (a `lax.map`) shed only the per-node gather working-set while still materialising the full `(..., n_nodes)` output — half a fix, and at large scale the residual stack can bind and even raise peak. Fold the weighted sum into a `lax.scan` carry instead: each block reads only its nodes and accumulates the running expectation, so neither the per-node gather nor the full node-stacked output is materialised. `0` (or a size >= the mesh) keeps the single fused vmap + reduction. The block reduction reorders the floating-point adds, so the value function matches the fused solve to tight numerical tolerance rather than bit-identically; the node-batch invariance test asserts `allclose` at rtol/atol 1e-9. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- New `docs/credits.md` (linked into the docs nav and the README), crediting the methods pylcm builds on (Carroll EGM; Iskhakov–Jørgensen–Rust–Schjerning DC-EGM; Dobrescu–Shanker FUES; Fella generalized EGM; the Druedahl / Druedahl–Jørgensen / Dobrescu–Shanker multidimensional-EGM methods behind the reserved upper-envelope backend), the discretization methods (Tauchen, Rouwenhorst, Kopecky–Suen, Fella–Gallipoli–Pan), the replicated example models shipped in pylcm (IJRS 2017, Mahler–Yum 2024), QuantEcon, and the open-source ecosystem (OSE, NumEconCopenhagen, akshayshanker, JeppeDruedahl, fediskhakov). - Add the missing BibTeX entries (Carroll 2006, Fella 2014, Dobrescu–Shanker FUES 2022 and RFC 2024, Druedahl 2021, Druedahl–Jørgensen 2017). - Fix the Mahler & Yum (2024) page range in references.bib: 1697–1733 (Econometrica 92(5)), not 1307–1343. Every citation verified against the journal/DOI or the authors' repositories. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
pylcm's Fast Upper-Envelope Scan was adapted from the `OpenSourceEconomics/upper-envelope` package (Apache-2.0, © The Upper-Envelope Authors) and since substantially modified. Both projects are Apache-2.0, so the derivation is permitted; this records the upstream attribution and the modification notice (Apache-2.0 §4(b)/(c)) in the `fues.py` header and the Credits page, which previously credited only the method's paper. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
pylcm's Fast Upper-Envelope Scan is adapted from OpenSourceEconomics/upper-envelope (Apache-2.0). Add a root `NOTICE` recording that derivation and the upstream copyright, and vendor the upstream license text verbatim at `licenses/upper-envelope-LICENSE`. The in-source attribution in `fues.py` (which ships in the wheel via `src/`) already carries the modification notice; these files add the conventional repo/sdist-level attribution. No other part of the codebase derives from third-party source — the DC-EGM kernel and the NEGM Phase-0 toys are written against pylcm's own API — so only the upper-envelope derivation is recorded. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move the pure spec-introspection helpers (discrete/passive/process state names, child state name, child discrete actions, child resources function and its arg names, regime-function concatenation) out of step.py into egm/regime_introspection.py. These read only a regime's static spec and hold no kernel state, so they form the dependency leaf that the kernel build, the continuation subsystem, and the kernel-scope checks all import from — breaking the otherwise-circular dependency between step.py and the scope checks that move next. Behavior-preserving pure move. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move the four `_find_unsupported_*` functions — the build-time checks that name features outside the kernel's current scope — out of step.py into egm/kernel_scope.py. They read the processed kernel-build context (period carry targets, qualified flat params, the regime's VInterpolationInfo), which the model-time `validation` module does not have, so they belong with the kernel build rather than with the model-construction validators. step.py keeps calling `_find_unsupported_feature` to install the raising step. Behavior- preserving pure move. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move the multi-target continuation machinery — target splitting, the per-child carry reader, per-row resources queries and gradients, the stochastic-node expectation, passive-state blending, discrete-choice aggregation (hard max or EV1 logsum), and the child-read build — out of step.py into egm/continuation.py. The core EGM step and the asset-row step keep calling `_get_child_carry_reader` / `_build_child_reads` / `get_egm_continuation_targets`, so step.py shrinks by ~870 lines and the textbook Euler inversion no longer sits in the same file as the multi-target, passive-state, taste-shock, and stochastic-node logic. Behavior-preserving pure move; the bound continuation contract is a follow-up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move the certifiable core algorithm — _EgmKernelPieces, _get_solve_one_combo, _compute_nodes_over_savings, _get_compute_node, _compute_constrained_candidates, _publish_V_and_carry_rows, and the constrained-offset constant — out of step.py into egm/step_core.py. It reads the expected continuation through continuation's per-target carry reader and depends on nothing in asset-row mode or the kernel orchestration. step.py keeps the orchestration (build, dispatch, combo map) and the asset-row path (extracted next). The one test importing _publish_V_and_carry_rows from step is repointed to step_core. Behavior- preserving pure move. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move the per-Euler-node solve — _get_solve_one_combo_asset_rows, its differentiated continuation closure _get_expected_continuation_value, and the scalar-query publisher _publish_node_V_and_policy — out of step.py into egm/asset_row.py. It maps step_core's single-post-state pipeline over the Euler grid and consumes continuation's per-target reader; step.py keeps only the orchestration (build, dispatch, combo map, feasibility, raising step). This completes the split: the textbook core (step_core) and the asset-row extension (asset_row) are now separate, certifiable modules. Behavior-preserving pure move; the streaming refine-to-query feature lands here next. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Collapse the multi-target continuation aggregation into `ContinuationPlan` (build-time statics) plus `bind_continuation`, which binds the plan to one combo pool and the next period's carries and returns a `savings -> (expected value, expected marginal)` callable. The textbook step (`_get_compute_node`) and the asset-row step (`_get_expected_continuation_value`) read only that bound callable instead of each re-deriving the regime-transition probabilities, child carry readers, and per-target aggregation. Behavior-preserving: the bound callable performs the identical aggregation, and it evaluates the probabilities and child reads from the combo pool *inside* the builder, so the asset-row Euler-slot gradient still carries the direct dP/da . EV terms (precomputing them outside the differentiated closure would silently drop them). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…the GridSearch rename. The re-rooted `SolverBuildContext` / `SolverKernels` are beartyped by the package claw, so under PEP 649 their field annotations resolve to real objects when a context is constructed at model build. `UserRegime` and `VInterpolationInfo` (whose module imports `lcm.regime`) close an import cycle through the `lcm.solvers` façade, so each is referenced via a two-form alias — precise for ty under `TYPE_CHECKING`, a bare container for the beartype claw at runtime. The other engine types import normally. The `BruteForce` → `GridSearch` rename had not reached the DC-EGM oracle test modules; propagate it, including the validation-message `match=` string. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`test_model_with_dcegm_solver_builds` asserts that selecting the DC-EGM solver builds the model rather than being rejected, but it placed the solver on the stock iskhakov `working_life` regime, which lacks the resources, post-decision, and inverse-marginal-utility functions the DC-EGM contract requires — so validation correctly rejected it. Point the test at `build_dcegm_model`, whose regimes satisfy the contract. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…r 2) Remove the residual solver-type fork in the backward-induction loop. The loop no longer branches `if regime.solution.egm_step is not None`; every regime — grid search or DC-EGM — exposes one uniform per-period adapter that wraps its shared jitted core, calls it with the solver's own argument layout, and assembles a `KernelResult(V_arr, carry, sim_policy)` outside JIT. The only branches left in the loop are on the optional generic outputs (`result.carry is not None` / `result.sim_policy is not None`), never on solver type. Contract (`_lcm/solution/contract.py`): - Add `KernelResult` (frozen dataclass) and `PeriodKernel` (runtime_checkable structural Protocol exposing `core`, `build_lower_args`, `with_fixed_params`, `__call__`). - Reshape `SolverKernels` → `SolutionKernels(period_kernels, continuation_template)`, dropping the `max_Q_over_a` / `egm_step` / `egm_carry_template` / `egm_reachable_targets` quartet. - Name the continuation channel solver-agnostically via `ContinuationPayload` (= `EGMCarry` today). Solvers (`_lcm/solution/solvers.py`): - `GridSearch`/`DCEGM.build_period_kernels` build per-period adapter closures (`_GridSearchPeriodKernel` / `_DCEGMPeriodKernel`) over the existing jitted core; the EGM core's reachable-carry filter and param projection move onto the DC-EGM adapter. Engine wiring: - `SolutionPhase` carries `period_kernels` + `continuation_template` (plus a `solves_via_dcegm` property); the cyclic `PeriodKernel` annotation uses the TYPE_CHECKING-precise / runtime-widened two-definition pattern so the beartype claw resolves the dataclass field at construction. - The terminal-continuation publisher is composed engine-side (`_TerminalCarryPeriodKernel` in regime_building/processing.py) as an output decorator, not inside `GridSearch`. - `model_processing._partial_fixed_params_into_regimes` binds fixed params via each adapter's `with_fixed_params`; `simulation/additional_targets.py` reads `solves_via_dcegm`. Compilation reuse and the loop's memory/host-offload/gc discipline are preserved: only the shared core is jitted and identity-deduped (by `id(Q_and_F)` / function identity), no adapter is jitted per period, and the solve loop's device eviction / carry-buffer release / reachable-carry subset logic is untouched (dispatch re-routed only). Add `test_period_kernels_sharing_a_config_reuse_one_compiled_core` asserting the distinct-compiled-core count. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
In asset-row mode the per-node solve refines a full NaN-padded upper envelope of static length n_pad and interpolates it at exactly one query (resources_at_node) to publish a scalar (V_node, policy_node), then discards it. Batched over combos x Euler-nodes, that n_pad scratch axis is the binder that OOMs large solves. Fold the single-query interpolation into the upper-envelope scan so the n_pad rows never materialize: - Extract `_interp_between_nodes` in interp.py — the pure two-bracket-node arithmetic plus the Hermite correction, shared by `interp_on_prepared_grid` and the streamed path. The existing interp unit tests pin it (behavior- preserving). This guarantees the streamed value cannot diverge from row-then-interp: only which two nodes differs. - Add `refine_to_bracket` + `QueryBracket` in fues.py — a new scan driver reusing `_inspect_candidate` verbatim, changing only the emission sink: each step's up-to-3 emitted points fold into an O(1) bracket-capture carry (first/second, rolling prev/last, lo = latest emitted with grid <= q, hi = first with grid > q, running counts), no [n_input,3] stack and no [n_pad] rows. Post-scan, the bracket reproduces clip(searchsorted(side="right"), 1, max(n_kept-1, 1)) node-for-node: the side="right" tie-break at a duplicated kink (right copy wins the lower slot), the below-first clamp (first, second), and the at-or-above-last clamp (second-last, last; single-live falls back to the NaN-padded slot like the row path). Geometry only — no utility, borrowing limit, or floor. - Add `publish_node_from_bracket` in asset_row.py — the scalar-bracket counterpart of `_publish_node_V_and_policy`: the shared `_interp_between_nodes` with the value Hermite slope = grad(utility) at the two bracket policies only, plus the constrained floor and the n_kept > n_pad overflow poison, preserved identically. Wire the asset-row node solve to refine_to_bracket -> publish_node_from_bracket and drop the n_pad envelope from that path. Scope: asset-row mode only. Single-post-state mode publishes the refined envelope AS its inter-period carry (queried later at many parent points), so step_core's single-post-state path keeps calling `refine` unchanged. The new equivalence test pins the streamed pair against refine + _publish_node_V_and_policy for the same candidates + query, to fp tolerance: smooth, kinked, multi-crossing, all-dead, single-live, overflow, and query below-first / above-last / exactly on a duplicated kink abscissa. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The backward-induction loop forced a `gc.collect()` every period to return the device pool from a rolled-off continuation carry (a registered pytree CPython frees only on a cyclic collection). A grid-search period rolls no such carry, so the collection bought nothing there while its fixed cost dominated the warm solve of small / many-period models — deteriorating those benchmarks by up to ~25x (the Iskhakov-et-al grid-search warm solve: 1.01s -> 0.35s). Gate the collection on the loop's own per-period carry output (`period_egm_carries`), empty exactly when no carry rolled. The backward- induction loop stays backend-agnostic — it does not fork on solver type or read `solution.solves_via_dcegm`; the clean separation of the solve backends is a separate design task. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Dobrescu-Shanker housing keeper (no-house-trade) regime is a plain 1-D DC-EGM model: liquid assets is the Euler state, consumption the continuous action, and housing a passive continuous state whose service flow `alpha * log(housing)` is read directly by utility. The cornerstone test locks in that the DC-EGM envelope-condition guard admits utility reading a *passive* continuous state (it forbids only the Euler state) by calling `validate_dcegm_regimes` directly on the finalized keeper regimes — no kernel trace, no solve. The full GPU solve is gated off the local box. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase-0 of the NEGM / multidimensional DC-EGM build (general 2-asset capability): - `negm_phase0/brute_feasibility_probe.py`: sweeps a 2-continuous-state (liquid + illiquid) + 2-continuous-action + income-shock lifecycle solved with BruteForce, reporting wall-clock and GPU peak. Finding: brute is single-solve feasible at fine grids (~4-9 s, <600 MB on a V100 up to 2.5 B combos/period) — the "brute breaks at Laibson scale" premise does not hold; NEGM is an accuracy/throughput/generality play, not a feasibility rescue. - `negm_phase0/kinked_toy_oracle.py`: the smallest 2-asset toy carrying the Laibson frictions (credit-card rate kink at a^X=0, withdrawal penalty at Iz=0, Z>=0 floor, u(C+iota*Z)), brute-solved as the V-parity oracle for a future NEGM prototype. - `negm_phase0/negm-phase0-findings.md`: brute-feasibility table, the NEGM nesting derivation with the P1 kink-candidate handling, build-vs-buy, and the gate recommendation (build for the general capability; reframe the motivation). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…del. Implements P2 of the Dobrescu–Shanker replication: the NEGM solver as a third `Solver` on the ABC seam, composing an inner 1-D `DCEGM` solve of the consumption-savings problem with an outer deterministic grid search over a durable/illiquid margin. - `NEGM(Solver)` in `_lcm/solution/solvers.py` (re-exported from `lcm.solvers` and `lcm`): frozen dataclass with `inner: DCEGM`, `outer_action`, `outer_post_decision`, `outer_grid`, optional `outer_no_adjustment_candidate`. `__post_init__` guards reject a stochastic outer grid and margins that coincide with the inner ones (`RegimeInitializationError`), mirroring DCEGM's `_fail_if_*` helpers. - `_NEGMPeriodKernel`: non-jitted outer adapter over the shared jitted inner DC-EGM core; sweeps the outer grid plus the per-node no-adjustment candidate, binds the outer post-decision into the regime's flat params per node, and collapses the outer axis by `max`. Static only (ty + prek green); not run. - `validate_negm_regimes` in `_lcm/egm/negm_validation.py`, wired into the same model-build path as `validate_dcegm_regimes`. Fail-loudly contract (`ModelInitializationError`, each message naming the alternative solver): no outer margin → use `DCEGM`; outer margin coincides with the inner margin → reject; coupled-2-Euler (outer post-decision feeds the inner Euler law / savings-stage pool, or a non-additively-separable utility cross-term) → use the 2-D EGM foundation; taste-shocked discrete choice → reject (ordering). - Kinked-toy NEGM model (`tests/test_models/negm_kinked_toy.py`) and the G1 parity test (`tests/solution/test_negm_kinked_toy.py`, skipped — GPU-only). - Unit tests: 6 config-guard cases, 8 contract-validator cases (construct and assert-raise, no solve). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The DS housing-keeper DC-EGM probe gains a brute-force (GridSearch) twin and a
DC-EGM-vs-brute value-parity test, mirroring tests/solution/test_egm_passive.
The brute twin solves the same keeper economics without the Euler/passive
split: housing is a regular fixed continuous state, consumption a grid-searched
action, the liquid-asset law of motion reads consumption directly, and a
borrowing constraint floors post-decision liquid assets at the borrowing limit
(the floor of the DC-EGM savings grid). build_working_regime, build_model, and
build_params gain a variant arg ("dcegm" | "brute").
The parity test asserts DC-EGM >= brute and allclose at the passive-
interpolation tolerance, excluding the lowest liquid-asset nodes where the
coarse consumption grid makes brute itself unreliable. It is skip-marked
(gpu-01 only) since it solves and OOMs the local box.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The splay/batch-size invariance assertions hard-coded float64 tolerances (rtol/atol 1e-12 for the combo axes, 1e-9 for the stochastic-node reduction), which the 32-bit CI job cannot meet: rescheduling the lax.map / reordering the weighted-sum adds shifts results at single-precision scale. Key the tolerance off conftest's X64_ENABLED, matching the established pattern in test_economic_validation / test_grids. Also stop test_egm_interp forcing dtype=jnp.float64 for its interpolation grid; under --precision=32 that raises a float64-truncated-to-float32 UserWarning that CI treats as an error. Use the canonical default float dtype instead. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The solve module's job is backward induction; name it for that. Renames src/_lcm/solution/solve_brute.py → backward_induction.py and tests/solution/test_solve_brute.py → test_backward_induction.py, updates every import (model.py, simulation/compile.py, test_beartype_claw.py), the two test-function names, and the doc/docstring references. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…oracle Add a dense host VFI oracle (`tests/solution/_ds2024_housing_vfi_oracle.py`) that mirrors the NEGM nest and searches the free-keep level `h(1-delta)` as an explicit candidate alongside the outer house grid. At delta>0 that level is off the house grid, so the on-grid brute twin cannot represent it — the oracle is the only valid reference for the depreciating keeper. Two interior tests: - the oracle reproduces the brute solve at delta=0 (pins its economics); - the NEGM solve at delta=0.10 reproduces the oracle (mean interior |dV| ~0.08, the same grid-resolution agreement as the delta=0 brute pair) — the keeper's depreciated hold is faithful. Exclude the oracle helper from the test-naming hook, matching the other tests/solution/_*_oracle.py helpers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…durable A reviewer flagged that `_build_coh_shift_function` lifts adjusters using the identity keeper reference (`next = durable`) rather than the keeper core's no-adjustment level `keep(durable) = durable(1-delta)`, which the outer-envelope design doc says should be the credited-cost reference. Switching to `keep(durable)` in isolation is the design-exact shift, but a differential VFI-oracle test shows it *regresses* DS-2024 delta=0.10: the corrected lift places each adjuster at a coh the fixed keeper-grid envelope reads by extrapolation, and the one-island-per-adjuster splice over-counts, moving the converging ~0.08 interior agreement to a plateaued ~0.18. The identity reference and the island splice are tuned together; the shipped solver converges (mean 0.0825 -> 0.0759 as n_grid 10 -> 18). Correcting both belongs with the segment-topology outer-envelope redesign, so document the coupling here rather than ship a regression. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Australia-2015-16 capital-income tax brackets are not printed in the DS-2026 paper text (only Figure 9); they are transcribed from the co-authors' replication repo. Record the exact provenance so upstream drift is detectable: GitHub akshayshanker/FUES, examples/housing_renting/config_HR/STD_RES_SETTINGS_4_TAXES/ master.yml, pinned at commit 00961e0b (blob 41661779), and a sha256[:16] checksum (4e8d1bf0748f6933) of the resolved (LOWER, OFFSET, RATE) arrays. All 10 brackets verified to match the source tax_table.brackets exactly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The GPU peak-memory tracker spawns `python -m benchmarks.asv._gpu_mem` with cwd=_PROJECT_ROOT, but _PROJECT_ROOT resolved to the benchmarks/ directory, not the repo root, so the subprocess could not import `benchmarks` and every GpuPeakMem benchmark failed with `ModuleNotFoundError: No module named 'benchmarks'` (the whole benchmark-pr workflow red). The file moved one level deeper in 7dc5887 (benchmarks/ -> benchmarks/asv/) but the `.parent.parent` chain was not updated. Point it at the true repo root (three parents up). Verified locally: `python -m benchmarks.asv._gpu_mem` resolves from the corrected cwd and fails only on missing argv, no longer on import. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
W1 made `inverse_marginal_utility` optional: when a regime omits it, EGM derives the inverse numerically (`numeric_inverse.py`; validation.py:298 documents this, and tests/solution/test_iegm_wiring.py covers a model solving without it). So removing that function is no longer a contract violation, and the negative case in test_dcegm_contract_violation_raises stopped raising — failing `main` deterministically on every platform. Drop the now-contradictory case; the iEGM path's positive coverage already lives in test_iegm_wiring.py. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The DC-EGM solve/oracle battery added on this branch makes the serial suite ~86 min on CI runners. Bake `-n 4` into the tests / tests-32bit / tests-with-cov tasks so every CI job (and local `pixi run tests`) distributes across 4 workers; `--dist loadfile` (already in addopts) keeps each file on one worker. Set `XLA_PYTHON_CLIENT_PREALLOCATE=false` on the tasks — a no-op on CPU, but required on GPU so the 4 worker processes don't each preallocate ~75% of the device and OOM; they grow to their actual small per-test need, and loadfile keeps the one memory-heavy file (Mahler-Yum) isolated to a single worker. Verified locally: 4/4 workers spin up and pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The state-action-param bundle threaded through `Q_and_F`, `max_Q_over_a`, and their `_transform_and_average` / `_invert_joint` helpers carries every flat param, including lookup-table params represented as `MappingLeaf` / `SequenceLeaf` (tax schedules, pension accrual tables). The bundle was annotated `FloatND | IntND | BoolND`, which excludes those leaf types, so the package beartype claw rejected a `MappingLeaf` param whenever its O(1) sampling happened to draw that key — an intermittent failure independent of the model's correctness. Widen the bundle annotation to the canonical `_ParamsLeaf` union. The two concrete taste-shock extractions (`taste_shocks__scale`, `taste_shock_key`) are narrowed back at their call sites with `cast`, since `logsum_and_softmax` and `draw_taste_shock_noise` require the scalar/array types. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…iles) The single self-hosted GPU runner is starved of host CPU/memory when four xdist workers each AOT-compile the DC-EGM oracle suite in parallel, and the runner loses communication with the server (~60 min, then death) instead of finishing the suite in roughly a quarter of that serially. Append `-n0` to the two GPU steps so pytest-xdist's last `-n` wins and the suite runs in-process. The CPU matrix keeps `-n 4`, where the four workers are a clear speedup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The macOS runner has 3 vCPUs and 7 GB of RAM; four xdist workers each AOT-compiling the DC-EGM solve/simulate/oracle tests peak past its memory and swap-thrash the suite to ~4 hours. A `pytest_collection_modifyitems` hook marks that battery `slow` (all of `tests/solution/`, plus the DC-EGM-family modules elsewhere), and the macOS matrix leg runs `-m "not slow"`. Linux and the GPU runner still run the full suite, so the platform-independent kernel stays covered; macOS keeps the lighter engine/regime/grid/simulation tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The committed `pixi.lock` is written in the format pixi 0.71.x emits; pixi 0.70.1 cannot parse it and aborts every job at install with `expected a string, found table`. Pin the `setup-pixi` steps to the version that produced the lock so the frozen install succeeds. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`test_simulate_using_raw_inputs` calls `simulate` directly, bypassing the public `Model.simulate` canonicalization, so its `flat_params` must already carry canonical dtypes. `jnp.asarray(0)` is `int64` under x64, but the params-leaf contract is `int32`; `_invert_joint`'s beartype check rejected it whenever its O(1) sampling drew that key. Pin the leaf to `int32` — the dtype `cast_params_to_canonical_dtypes` produces on the public path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reconcile the two independent non-EU implementations that collided: - Adopt main's first-class `CertaintyEquivalent` seam (`resolve_certainty_ equivalent`, `ce.transform`/`ce.inverse`) throughout `Q_and_F.py`, dropping feat/dcegm's superseded `value_transform`/`inverse_value_transform` pair (`_build_continuation_operator`, `_transform_and_average`, `_invert_joint`, `_apply_continuation_operator`) and its test. DC-EGM keeps its own structure (`_get_deterministic_transitions` durable-carry, co-map, extended `_get_U_and_F`); the auto-merge's double-transform is removed. - Thread `certainty_equivalent` through `processing.py` solve/simulate builders alongside the existing DC-EGM params (`solver`, `has_taste_shocks`). - Repoint the DC-EGM custom-H admissibility check to main's default aggregator: `_default_H` (lcm.regime) → `H_linear` (lcm.temporal_aggregation), the object `finalize.py` now injects, so the identity check stays correct. - Keep feat/dcegm's `period_kernels`/`build_lower_args` kernel API in the distributed HLO test (main's `max_Q_over_a` no longer exists on this branch). ty + ruff clean; build-time DC-EGM validation, certainty-equivalent, Q_and_F, custom-aggregator, and simulate-CE tests pass on CPU. DC-EGM×EZ solve behavior must be confirmed on GPU CI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The 32-bit CI job failed 25 assertions whose tolerances (rtol down to 1e-12, atol down to 1e-12, decimal=10) are only meaningful at 64-bit. Follow the existing `X64_ENABLED` convention: keep the 64-bit values and use float32-eps-scaled bounds (typically 1e-5, with at least an order of magnitude of margin over the observed 32-bit errors) otherwise. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Near the optimum the simulate argmax is nearly flat: the top-2 consumption-node values differ by less than the solve's cross-backend variation, which enters at kink-adjacent nodes of the degenerate last-alive-period EGM step (zero terminal continuation) and reaches a few 1e-3 in V. Different backends and CPU microarchitectures therefore legitimately settle on adjacent consumption-grid nodes; the windows CI runner deterministically picked the neighbor of the pinned node. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two memory pathologies from running the DC-EGM battery in long-lived pytest processes: - ubuntu: JAX compile caches grow a worker's resident memory with every distinct compiled program, and four parallel workers running the full suite exhaust the 16 GB runner (the agent receives a shutdown signal mid-suite, seen twice at ~75%). Run the coverage job in three sequential invocations (fast set -n 4, then the slow battery in two fresh -n 2 processes) so each phase's memory is released; coverage is appended across them. - GPU: the Mahler & Yum solve kernels only fit the T4 when XLA fully fuses the max-over-actions reduction. Compiled late in a process that has run the whole battery, the executable came out unfused, with a 38.4 GB scratch arena (half the state-action product in float64) that can never fit a 16 GB device. The module now runs first, in its own process, pinning the fused compile; the identical program lowers to a 0.5 GB arena in a fresh process. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The autoupdate commit dropped the lock file along with the rev bumps; every frozen CI install needs it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…py use - Widen the one-hot regime-transition wrapper's parameter annotations to the TransitionFunction protocol's union (parameters are contravariant, so the narrower `FloatND | IntND | int` was not assignable). - Replace the deprecated `scipy.interpolate.interp1d(kind="cubic")` with `make_interp_spline(k=3)` — the same not-a-knot cubic spline, verified bit-identical on the knot layout the example uses. - Pass `gauss_hermite` / `n_std` explicitly instead of through a heterogeneously-typed kwargs dict in the IID-process moment tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ty now runs as the official `astral-sh/ty-pre-commit` hook, so one `prek run --all-files` covers all checks: - The hook passes `--no-project` so uv neither creates a `.venv` nor resolves deps; ty resolves third-party imports from the pixi env named in `[tool.ty] environment.python` (the `type-checking` env, which keeps the stub packages but no longer carries the ty binary or a `ty` task). - pre-commit.ci skips the hook (no pixi envs, no network at hook runtime); the run-ty CI job runs it against the `type-checking` env. - Docs and agent instructions now name `prek run ty --all-files`. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Brings the ty-via-prek type-checking section, the module overhaul (distilled modules, numpy.md folded away), and the pandas provenance rule. All @-references from this repo's AGENTS.md still resolve. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
actions/checkout v6 -> v7, prefix-dev/setup-pixi v0.9.6 -> v0.10.0, pixi v0.71.2 -> v0.72.0, codecov/codecov-action v6.0.1 -> v7.0.0. actions/setup-python stays on the v6 major and pypa/gh-action-pypi-publish on its rolling release/v1 tag. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Addresses the findings from the PR #390 code review. Correctness: - step_core: NaN-poison the carry marginal-utility row on envelope overflow, matching the value rows, so an overflowed period cannot feed the parent a finite-but-wrong Hermite slope past the NaN diagnostics. - numeric_inverse: require finite bracket marginals in the log_well_defined gate, so a +inf marginal utility (steep CRRA at a near-zero c_lower) fails loud instead of running the log path on a non-finite endpoint. - FUES upper envelope (#387): scan exhaustively by default (n_points_to_scan=None -> every candidate). A bounded window silently accepts dominated points when more than the window's worth of off-segment candidates interleave between two points of one segment. An explicit finite width remains available as a speed-vs-correctness opt-in. The F4 strict xfail becomes a passing test; a companion test pins the bounded-mode tradeoff. Cleanup and documentation: - regime_template: delete the dead value_transform/inverse_value_transform special-case orphaned by the certainty-equivalent merge. - validation: correct the module and helper docstrings that overclaimed the grid rules (batch_size is honored; only distributed is rejected) and that wrongly listed inverse_marginal_utility as required. - asset_row: docstring "weakly ascending" -> "strictly increasing" to match the enforced resources-monotonicity check. - continuation: document why a device-sharded carry into a DCEGM parent is unsupported, that three existing rules already make it unconstructible, and concrete ideas for lifting the restriction. A regression test pins the fence. Tests: ty, ruff, and the EGM/DCEGM/simulation suites pass (525 passed, 3 skipped); FUES parity suites green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0123hk3uzBgBeZown4eBBQ2C
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the DC-EGM solver (discrete-continuous endogenous grid method) behind the
SolverABC seam. It inverts the Euler equation on an exogenous end-of-period gridinstead of grid-searching the continuous action, with a FUES upper envelope to drop
dominated candidates. Selected per regime via
solver=DCEGM(...);GridSearch(thedefault) is untouched, and the engine dispatches polymorphically — no fork on solver
type.
Commit structure (reviewable in steps)
envelope, the closed-form constrained segment, multi-target continuation, and
asset-row mode; wired through the ABC
DCEGM.validate/build_period_kernels.lax.scanblocks — a memory knob for the childstochastic-node mesh.
an acknowledgments page.
step.py(5 pure-move commits) —regime_introspection→kernel_scope→continuation→step_core→asset_row, taking the monolith from~2900 to ~800 lines so the simple single-post-state path reads cleanly apart from the
asset-row and continuation machinery.
savings → (expected_value, expected_marginal)callable (ContinuationPlan).package claw, so cyclic field types use the two-definition alias pattern (precise for
ty, bare container at runtime); plus a valid-DCEGM seam build test.
PeriodKernel/KernelResultadapter; the solve loopcalls one kernel and branches only on optional generic outputs, not on solver type.
FUES scan (
refine_to_bracket+publish_node_from_bracket), removing then_padintra-node envelope scratch. Single-post-state still publishes the full refined
envelope as its carry, unchanged.
Solvers and accuracy work built on the DC-EGM seam
On top of the kernel, this branch adds the prime-time solvers and the Dobrescu–Shanker
benchmark reproductions:
(
DCEGM(upper_envelope=...)), plus the 2-D RFC/G2EGM two-asset foundation for theDS-2024 pension comparison.
EGM-FUES, discrete-housing-with-tax); full paper-scale matrix run on gpu-01.
tests/solution/_branch_aware_vfi_oracle.py): pinsthe App.2 (S,s) EGM-vs-VFI gap to a comparator-ordering term that is identically zero
on the scored interior, correcting the earlier attribution; the tolerated interior gap
is DC-EGM durable-corner discretization. Standard-mode oracle == brute to 8e-5.
ds2024_housing*.py,benchmarks/ds_replication/ds2024_housing_comparison.py): NEGM (nested continuoushousing) vs RFC/FUES (discrete-housing DC-EGM, the source's per-housing-column 1-D
rooftop cut). Reproduces the paper headline — RFC runtime < NEGM runtime — against a
grid-search VFI oracle (faithful at
delta = 0; the paper'sdelta = 0.10keeperdepreciation awaits a shared housing-axis carry extension).
Validation
-n 4, on a free GPU): 1405 passed, 15skipped, 2 xfailed. Covers the DC-EGM-vs-brute value-function-parity battery, the
19-case refine-to-query unit equivalence (streamed ≡ row-then-interp), and the Layer-2
distinct-compiled-cores regression.
tyandprekclean.Pending (does not block reviewing the engine)
delta = 0.10fidelity: the keeper depreciates the held stock offthe housing grid; the strict-identity NEGM keeper needs a housing-axis carry extension.
n_padruntime win is only visible atACA scale — the in-suite oracles are too small to exercise it; handed to the aca side.
experiment this work unlocks, not settled here.
🤖 Generated with Claude Code